Simulation-based Learning of Optimal Multimodal Presentation Strategies from Wizard-of-Oz data

We address two problems in the field of automatic optimization of dialogue strategies: learning effective dialogue strategies when no initial data or system exists, and optimising dialogue management (DM) and Natural Language Generation (NLG) decisions in an integrated fashion. We use Reinforcement Learning (RL) to learn multimodal information presentation strategies through interaction with a simulated environment which is “bootstrapped” from small amounts of Wizard-of-Oz (WOZ) data. This use of WOZ data allows development of optimal strategies for domains where no working prototype is available. For information seeking dialogues, Dialogue Management and NLG are two closely interrelated problems: the decision of when to present information depends on the available options for how to present them, and vice versa. We therefore formulate the problem as a hierarchy of joint learning decisions which are optimised together. To evaluate, we compare the RL-based strategy against a supervised learning (SL) strategy which mimics the (human) wizards’ policies from the original data. This comparison allows us to measure relative improvement over the training data. Our results show that RL significantly outperforms SL: the RL-based policy gains on average 50-times more reward when tested in simulation. In related work we evaluate the strategies with real users [16].